The Interchangeability of Learning Rate and Gain in

The backpropagation algorithm is widely used for training multilayer neural networks In this publication the gain of its activation function s is investigated In speci c it is proven that changing the gain of the activation function is equivalent to changing the learning rate and the weights This simpli es the backpropagation learning rule by eliminating one of its parameters The theorem can be extended to hold for some well known variations on the backpropagation algorithm such as using a momentum term at spot elimination or adaptive gain Furthermore it is successfully applied to compensate for the non standard gain of optical sigmoids for optical neural networks